Toward Evaluation of Writing Style: Overly Repetitious Word Use

essay scoring systems have been made available (PEG;Page 1966; e-rater®Burstein et al., 1998; Intelligent Essay AssessorTm;Foltz, Kintsch, and Landauer 1998; and, Intellimetric TM ; Elliot, 2003). In addition, based on the demands of users of the automated scoring technology, tools have been developed that perform more detailed evaluations of student writing. One such application is Critique Writing Analysis Tools. Critique and erater are embedded in a broader writing instruction application, Criterion sm Online Essay Evaluation (see http://www.etstechnologies.com). Critique performs a number of evaluations on a student essay related to errors in grammar (Chodorow and Leacock, 2000), usage, and mechanics, comments on style, and analysis of essay-based discourse (organization and development) (Burstein et al, 2001 and Burstein and Marcu, 2003, and Burstein, Marcu and Knight, forthcoming). Many of these capabilities use machinelearning approaches to model each particular kind of analysis. To develop such tools requires large sets of human annotated data, where judges have annotated information required to train a system to evaluate a particular kind of essay characteristic. For example, to build a capability to identify sentence fragments, a corpus of essay data needs to be annotated for this kind of ungrammatical sentence. A capability exists that identifies essaybased discourse elements in essays, for example, thesis statements, and conclusions. To do this, human judges annotated a corpus of essays for these particular kinds of discourse elements. Abstract